[Chapter 24] 24.2 Discovering an Intruder
Author: Simson Garfinkel & Gene Spafford
Date: 1999-02-04T00:18:42Z

#

| Chapter 24 Discovering a Break-in |

## 24.2 Discovering an Intruder

There are several ways you might discover a break-in:

- Catching the perpetrator in the act. For example, you might see the superuser logged in from a dial-up terminal when you are the only person who should know the superuser password.

- Deducing that a break-in has taken place based on changes that have been made to the system. For example, you might receive an electronic mail message from an attacker taunting you about a security hole, or you may discover new account entries in your /etc/passwd file.

- Receiving a message from a system administrator at another site indicating strange activity at his or her site that has originated from an account on your machine.

- Strange activities on the system, such as system crashes, significant hard disk activity, unexplained reboots, minor accounting discrepancies,[1] or sluggish response when it is not expected (Crack may be running on the system).

  > [1] See Cliff Stoll's The Cuckoo's Egg for the tale of how such a discrepancy led to his discovery of a hacker's activities.

There are a variety of commands that you can use to discover a break-in, and some excellent packages, such as Tiger and Tripwire, described elsewhere in this book. Issue these commands on a regular basis, but execute them sporadically as well. This introduces a factor of randomness that can make perpetrators unable to cover their tracks. This principle is a standard of operations security: try to be unpredictable.

### 24.2.1 Catching One in the Act

The easiest way to catch an intruder is by looking for events that are out of the ordinary. For example:

- A user who is logged in more than once. (Many window systems register a separate login for each window that is opened by a user, but it is usually considered odd for the same user to be logged in on two separate dial-in lines at the same time.)

- A user who is not a programmer but who is nevertheless running a compiler or debugger.

- A user who is making heavy and uncharacteristic use of the network.

- A user who is initiating many dialout calls.

- A user who does not own a modem logged into the computer over a dial-in line.

- A person who is executing commands as the superuser.

- Network connections from previously unknown machines, or from sites that shouldn't be contacting yours for any reason.

- A user who is logged in while on vacation or outside of normal working hours (e.g., a secretary dialed in by phone at 1:00 a.m. or a computer science graduate student working at 9:00 a.m.).

UNIX provides a number of commands to help you figure out who is doing what on your system. The `finger, users, whodo, w,` and who commands all display lists of the users who are currently logged in. The `ps` and `w` commands help you determine what any user is doing at any given time; `ps` displays a more comprehensive report, and `w` displays an easy-to-read summary. The `netstat` command can be used to check on current network connections and activity.

If you are a system administrator, you should be in the habit of issuing these commands frequently to monitor user activity. After a while, you will begin to associate certain users with certain commands. Then, when something out of the ordinary happens, you will have cause to take a closer look.

Be aware, however, that all of these commands can be "fooled" by computer intruders with sufficient expertise. For example, `w`, `users`, and `finger` all check the /etc/utmp file to determine who is currently logged in to the computer. If an intruder erases or changes his entry in this file, these commands will not report the intruder's presence. Also, some systems fail to update this file, and some window systems do not properly update it with new entries, so even when the file is protected, it may not have accurate information.

As the `ps` command actually examines the kernel's process table, it is more resistant to subversion than the commands that examine the /etc/utmp file. However, an intruder who also has attained superuser access on your system can modify the `ps` command or the code in the system calls it uses so that it won't print his or her processes. Furthermore, any process can modify its `argv` arguments, allowing it to display whatever it wishes in the output of the `ps` command. If you don't believe what these commands are printing, you might be right!

### 24.2.2 What to Do When You Catch Somebody

You have a number of choices when you discover an intruder on your system:

1. Ignore them.

2. Try to contact them with `write` or `talk`, and ask them what they want.

3. Try to trace the connection and identify the intruder.

4. Break their connection by killing their processes, unplugging the modem or network, or turning off your computer.

5. Contact a FIRST (Forum of Incident Response and Security Teams) team (e.g., the CERT-CC[2]) to notify them of the attack.

   > [2] See Appendix F, Organizations, for a complete list of FIRST teams.

What you do is your choice. If you are most inclined towards option #1, you probably should reread Chapter 1, Introduction, and Chapter 2, Policies and Guidelines, . Ignoring an intruder who is on your system essentially gives him or her free reign to do harm to you, your users, or others on the network.

If you choose option #2, keep a log of everything the intruder sends back to you, then decide whether to pursue one of the other options. Options #3 and #4 are discussed in the sections "Tracing a Connection" and "Getting Rid of the Intruder" that follow.

> NOTE: Some intruders are either malicious in intent, or extremely paranoid about being caught. If you contact them, they may react by trying to delete everything on disk so as to hide their activities. If you try to contact an intruder who turns out to be one of these types, be certain you have a set of extremely current backups!

If the intruder is logged into your computer over the network, you may wish to trace the connection first, because it is usually much easier to trace an active network connection than one that has been disconnected.

After the trace, you may wish to try to communicate with the intruder using the `write` or `talk` programs. If the intruder is connected to your computer by a physical terminal, you may wish to walk over to that terminal and confront the person directly (then again, you might not!).

> NOTE: Avoid using `mail` or `talk` to contact the remote site if you are trying to trace the connection without tipping off the intruder, because the remote root account on the remote site may be compromised. Try to use the telephone, if at all possible, before sending email. If you must send a message, send something innocuous, such as "could you please give me a call: we are having problems with your mailer." Of course, if somebody calls you, verify who they are.

### 24.2.3 Monitoring the Intruder

You may wish to monitor the intruder's actions to figure out what he is doing. This will give you an idea if he is modifying your accounting database, or simply rummaging around through your users' email.

There are a variety of means that you can use for monitoring the intruder's actions. The simplest way is to use programs such as `ps` or `lastcomm` to see which processes the intruder is using.

Depending on your operating system, you may be able to monitor the intruder's keystrokes using programs such as `ttywatch` or `snoop`. These commands can give you a detailed, packet-by-packet account of information sent over a network. They can also give you a detailed view of what an intruder is doing. For example:

>     # snoop
>     asy8.vineyard.net -> next         SMTP C port=1974
>     asy8.vineyard.net -> next         SMTP C port=1974 MAIL FROM:<dfddf@vin
>             next -> asy8.vineyard.net SMTP R port=1974 250 <dfddf@vineyard.
>     asy8.vineyard.net -> next         SMTP C port=1974
>     asy8.vineyard.net -> next         SMTP C port=1974 RCPT TO:<vdsalaw@ix.
>             next -> asy8.vineyard.net SMTP R port=1974 250 <vdsalaw@ix.netc
>     asy8.vineyard.net -> next         SMTP C port=1974
>     asy8.vineyard.net -> next         SMTP C port=1974 DATA\r\n
>             next -> asy8.vineyard.net SMTP R port=1974 354 Enter mail, end

In this case, an email message was intercepted as it was sent from asy8.vineyard.net to the computer next. As the above example shows, these utilities will give you a detailed view of what people on your system are doing, and they have a great potential for abuse.

You should be careful with the tools that you install on your system, as these tools can be used against you, to monitor your monitoring. Also, consider using tools such as `snoop` on another machine (not the one that has been compromised). Doing so lessens the chance of being discovered by the intruder.

### 24.2.4 Tracing a Connection

The `ps`, `w`, and `who` commands all report the terminals to which each user (or each process) is attached. Terminal names like /dev/tty01 may be abbreviated to tty01 or even to 01. Generally, names like tty01, ttya, or tty4a represent physical serial lines, while names that contain the letters p, q, or r (such as ttyp1) refer to network connections (virtual ttys, also called pseudo-terminals or ptys).

If the intruder has called your computer by telephone, you may be out of luck. In general, telephone calls can be traced only by prior arrangement with the telephone company. However, many telephone companies offer special features such as CALL*TRACE and CALLER*ID (CNID), which can be used with modem calls as easily as with voice calls. If you have already set up the service and installed the appropriate hardware, all you need to do is activate it. Then you can read the results.

If the intruder is logged in over the network, you can use the `who` command to determine quickly the name of the computer that the person may have used to originate the connection. Simply type `who`:

>     % who
>     orpheus  console  Jul 16 16:01
>     root     tty01    Jul 15 20:32
>     jason    ttyp1    Jul 16 18:43 (robot.ocp.com)
>     devon    ttyp2    Jul 16 04:33 (next.cambridge.m)
>     %

In this example, the user orpheus is logged in at the console, user root is logged on at tty01 (a terminal connected by a serial line), and jason and devon are both logged in over the network: jason from robot.ocp.com, and devon from next.cambridge.ma.us.

Some versions of the `who` command display only the first 16 letters of the hostname of the computer that originated the connection. (The machine name is stored in a 16-byte field in /etc/utmp; some versions of UNIX store more letters.) To see the complete hostname, you may need to use the `netstat` command (described in Chapter 16, TCP/IP Networks). You will also have to use `netstat` if the intruder has deleted or modified the /etc/utmp file to hide his presence. Unfortunately, netstat does not reveal which network connection is associated with which user. (Of course, if you have the first 16 characters of the hostname, you should be able to figure out which is which, even if /etc/utmp has been deleted. You can still use `netstat` and look for connections from unfamiliar machines.) Luckily, most modern versions of UNIX, including SVR4, report the entire machine name.

Let's say that in this example we suspect Jason is an intruder, because we know that the real Jason is at a yoga retreat in Tibet (with no terminals around). Using `who` and `netstat`, we determine that the intruder who has appropriated Jason's account is logged in remotely from the computer robot.ocp.com. We can now use the `finger` command to see which users are logged onto that remote computer:

>     % finger @robot.ocp.com
>     [robot.ocp.com]
>     Login     Name                  TTY Idle       When
>     olivia    Dr. Olivia Layson      co  12d   Sun 11:59
>     wonder    Wonder Hacker          p1        Sun 14:33
>     %

Of course, this method doesn't pin the attacker down, because the intruder may be using the remote machine only as a relay point. Indeed, in the above example, Wonder Hacker is logged into ttyp1, which is another virtual terminal. He's probably coming from another machine, and simply using robot.ocp.com as a relay point. You would probably not see a username like Wonder Hacker. More likely, you would only see an assorted list of apparently legitimate users and have to guess who the attacker is. Even if you did see a listing such as that, you can't assume anything about who is involved. For instance, Dr. Layson could be conducting industrial espionage on your system, using a virtual terminal (e.g., xterm) that is not listed as a logged in session!

If you have an account on the remote computer, log into it and find out who is running the `rlogin` or `telnet` command that is coming into your computer. In any event, consider contacting the system administrator of that remote computer and alert him or her to the problem.

#### 24.2.4.1 Other tip-offs

There are many other tip-offs that an intruder might be logged onto your system. For example, you may discover that shells are running on terminals that no one seems to be logged into at the moment. You may discover open network connections to machines you do not recognize. Running processes may be reported by some programs but not others.

Be suspicious and nosy.

#### 24.2.4.2 How to contact the system administrator of a computer you don't know

Often, you can't figure out the name and telephone number of the system administrator of a remote machine, because UNIX provides no formal mechanism for identifying such people.

One good way is to contact the appropriate incident response team for the designated security person at the organization. Another way to find out the telephone number and email address of the remote administrator is to use the `whois` command to search the Network Information Center (NIC) registration database. If your system does not have a `whois` command, you can simply telnet to the NIC site. Below is an example of how to find the name and phone number of a particular site administrator.

The NIC maintains a database of the names, addresses, and phone numbers of significant network users, as well as the contact people for various hosts and domains. If you can connect to the host whois.internic.net via telnet, you may be able to get the information you need. Try the following:

1. Connect to the host whois.internic.net via telnet.

2. At the > prompt, type `whois`.

3. Try typing `host robot.ocp.com` (using the name of the appropriate machine, of course). The server may return a record indicating the administrative contact for that machine.

4. Try typing `domain ocp.com` (using the appropriate domain). The server may return a record indicating the administrative contact for that domain.

5. When done, type `quit` to disconnect.

Here is an example, showing how to get information about the domain whitehouse.gov:

>     % telnet whois.internic.net
>     Trying 198.41.0.6 ...
>     Connected to rs.internic.net.
>     Escape character is `^]'.

>     SunOS UNIX 4.1 (rs1) (ttyp1)

>     ***********************************************************************
>     * -- InterNIC Registration Services Center  --
>     *
>     * For wais, type:                    WAIS <search string> <return>
>     * For the *original* whois type:     WHOIS [search string] <return>
>     * For referral whois type:           RWHOIS [search string] <return>
>     *
>     * For user assistance call (703) 742-4777
>     # Questions/Updates on the whois database to HOSTMASTER@internic.net
>     * Please report system problems to ACTION@internic.net

>     ***********************************************************************

>     Please be advised that use constitutes consent to monitoring
>     (Elec Comm Priv Act, 18 USC 2701-2711)

>     Cmdinter Ver 1.3 Tue Oct 17 21:51:53 1995 EST
>     [xterm] InterNIC > whois

>     Connecting to the rs Database . . . . . .
>     Connected to the rs Database
>     Whois: whitehouse.gov
>     Executive Office of the President USA (WHITEHOUSE-HST) WHITEHOUSE.GOV
>                                                                    198.137.240.100
>     Whitehouse Public Access (WHITEHOUSE-DOM)                       WHITEHOUSE.GOV
>     Whois: whitehouse-dom
>     Whitehouse Public Access (WHITEHOUSE-DOM)
>        Executive Office of the President USA
>        Office of Administration
>        Room NEOB 4208
>        725 17th Street NW
>        Washington, D.C. 20503
>        Domain Name: WHITEHOUSE.GOV

>        Administrative Contact:
>           Fox, Jack S.  (JSF)  fox_j@EOP.GOV
>           (202) 395-7323
>        Technical Contact, Zone Contact:
>           Ranum, Marcus J.  (MJR)  mjr@BSDI.COM
>           (410) 889-6449

>        Record last updated on 17-Oct-94.
>        Record created on 17-Oct-94.

>        Domain servers in listed order:

>        GATEKEEPER.EOP.GOV           198.137.241.3
>        ICM1.ICP.NET                 192.94.207.66

>     Whois: quit
>     [xterm] InterNIC > quit

>     Tue Oct 17 21:55:30 1995 EST
>     Connection closed by foreign host.
>     %

In addition to looking for information about the host, you can look for information about the network domain. You may find that technical contacts are more helpful than administrative contacts. If that approach fails, you can attempt to discover the site's network service provider (discovered by sending packets to the site using `traceroute`) and call them to see if they have contact information. Even if the site's network service provider will tell you nothing, he or she will often forward messages to the relevant people. In an emergency, you can call the organization's main number and ask the security guard to contact the computer center's support staff.

If you are attempting to find out information about a U.S. military site (the hostname ends in .mil), you need to try the whois command at nic.ddn.mil instead of the one at the InterNIC.

Another thing to try is to `finger` the root account of the remote machine. Occasionally this will produce the desired result:

>     % finger root@robot.ocp.com
>     [robot.ocp.com]
>     Login name: root in real life: Joel Wentworth
>     Directory: / Shell: /bin/csh
>     Last login Sat April 14, 1990 on /dev/tty
>     Plan:
>     For information regarding this computer, please contact
>     Joel Wentworth at 301-555-1212

More often, unfortunately, you'll be given useless information about the root account:

>     % finger root@robot.ocp.com
>     [robot.ocp.com]
>     Login name: root in real life: Operator
>     Directory: / Shell: /bin/csh
>     Last login Mon Dec. 3, 1990 on /dev/console
>     No plan

In these cases, you can try to figure out who is the computer's system administrator by connecting to the computer's `sendmail` daemon and identifying who gets mail for the root or postmaster mailboxes:

>     % telnet robot.ocp.com smtp
>     Trying...
>     Connected to robot.ocp.com
>     Escape character is "^]".
>     220 robot.ocp.com Sendmail NeXT-1.0 (From Sendmail 5.52)/NeXT-1.0   ready at Sun, 2 Dec 90 14:34:08 EST
>     helo mymachine.my.domain.com
>     250 robot.ocp.com Hello mymachine.my.domain.com, pleased to meet you
>     vrfy postmaster
>     250 Joel Wentworth <jw>
>     expn root
>     250 Joel Wentworth <jw>
>     quit
>     221 robot.ocp.com closing connection
>     Connection closed by foreign host.

You can then use the `finger` command to learn this person's telephone number.

Unfortunately, many system administrators have disabled their `finger` command, and the `sendmail` daemon may not honor your requests to verify or expand the alias. However, you may still be able to identify the contact person.

If all else fails, you can send mail to the " postmaster" of the indicated machine and hope it gets read soon. Do not mention a break-in in the message - mail is sometimes monitored by intruders. Instead, give your name and phone number, indicate that the matter is important, and ask the postmaster to call you. (Offering to accept collect calls is a nice gesture and may improve the response rate.) Of course, after you've phoned, find out the phone number of the organization you're dealing with and try phoning back - just to be sure that it's the administrator who phoned (and not the intruder who read your email and deleted it before it got to the administrator). You can also contact the folks at one of the FIRST teams, such as the CERT-CC. They have some additional resources, and they may be able to provide you with contact information.

### 24.2.5 Getting Rid of the Intruder

Killing your computer's power - turning it off - is the very quickest way to get an intruder off your computer and prevent him from doing anything else - including possibly further damage. Unfortunately, this is a drastic action. Not only does it stop the intruder, but it also interrupts the work of all of your legitimate users. It may also delete evidence you night need in court some day, delete necessary evidence of the break-in, such as running processes (e.g., `mailrace`), and cause the system to be damaged when you reboot because of the Trojaned startup scripts. In addition, the UNIX filesystem does not deal with sudden power loss very gracefully: pulling the plug might do significantly more damage than the intruder might ever do.

In some cases, you can get rid of an intruder by politely asking him or her to leave. Inform the person that breaking into your computer is both antisocial and illegal. Some computer trespassers have the motivation of a child sneaking across private property; they often do not stop to think about the full impact of their actions. However, don't bet on your intruder being so simplistic, even if he acts that way. (And keep in mind our warning earlier in this chapter.)

If the person refuses to leave, you can forcibly kill his or her processes with the `kill` command. Use the `ps` command to get a list of all of the user's process numbers, change the password of the penetrated account, and finally kill all of the attacker's processes with a single `kill` command. For example:

>     # ps -aux
>     USER  PID  %CPU  %MEM   VSIZE   RSIZE  TT  STAT   TIME  COMMAND
>     root 1434  20.1   1.4    968K   224K   01    R    0:00  ps aux
>     nasty 147   1.1   1.9   1.02M   304K   p3    S    0:07  - (csh)
>     nasty 321  10.0   8.7    104K   104K   p3    S    0:09  cat /etc/passwd
>     nasty 339   8.0   3.7   2.05M   456K   p3    S    0:09  rogue
>     ...
>     # passwd nasty
>     Changing password for nasty.
>     New password: rogue32
>     Retype new password: rogue32
>     # kill -9 147 321 339

You are well-advised to change the password on the account before you kill the processes - especially if the intruder is logged in as root. If the intruder is a faster typist than you are, you might find yourself forced off before you know it! Also bear in mind that most intruders will install a back door into the system. Thus, even if you change the password, that may not be sufficient to keep them off: you may need to take the system to single-user mode and check the system out, first.

As a last resort, you can physically break the connection. If the intruder has dialed in over a telephone line, you can turn off the modem - or unplug it from the back of the computer. If the intruder is connected through the network, you can unplug the network connector - although this will also interrupt service for all legitimate users. Once the intruder is off your machine, try to determine the extent of the damage done (if any), and seal the holes that let the intruder get in. You also should check for any new holes that the intruder may have created. This is an important reason for creating and maintaining the checklists described in Chapter 9, Integrity Management.

### 24.2.6 Anatomy of a Break-in

The following story is true. The names and a few details have been changed to protect people's jobs.

Late one night in November 1995, a part-time computer consultant at a Seattle-based firm logged into one of the computers that he occasionally used. The system seemed sluggish, so he ran the `top` command to get an idea of what was slowing down the system. The consultant noticed that a program called `vs` was consuming a large amount of system resources. The program was running as superuser.

Something didn't look right. To get more information, the programmer ran the ps command. That's when things got stranger still - the mysterious program didn't appear when ps was run. So the occasional system manager used the `top` command again, and, sure enough, the `vs` program was still running.

The programmer suspected a break-in. He started looking around the filesystem using the Emacs `dired` command and found the `vs` program in a directory called /var/.e. That certainly didn't look right. So the programmer went to his shell window, did a `chdir()` to the /var directory, and then did an `ls -a`. But the `ls` program didn't show the directory /var/.e. Nevertheless, the program was definitely there: it was still visible from the Emacs `dired` command.

The programmer was now pretty sure that somebody had broken into the computer. And the attack seemed sophisticated, because system commands appeared to have been altered to hide evidence of the break-in. Not wanting to let the break-in proceed further, the operator wanted to shut down the computer. But he was afraid that the attacker might have booby-trapped the /etc/halt command to destroy traces of the break-in. So before the programmer shut down the system, he used the `tar` command to make a copy of the directory /var/.e, as well as the directories /bin and /etc. As soon as the tar file was made, he copied it to another computer and halted the system.

The following morning, the programmer made the following observations from the tar file:

- Somebody had broken into the system.

- The program /bin/login had been modified so that anybody on the Internet could log into the root account by trying a special password.

- The /var/.e/vs program that had been left running was a password sniffing program. It listened on the company's local area network for users typing their passwords; these passwords were then sent to another computer elsewhere on the Internet.

- The program /bin/ls and /bin/ps had been modified so that they would not display the directory /var/.e.

- The inode creation dates and the modification times on the files /bin/ls, /bin/ps and /bin/login had been reset to their original dates before the modifications took place. The checksums for the modified commands (as computed with the `sum` command) matched those of the original, unmodified versions. But a comparison of the programs with a backup made the previous month revealed that the programs had been changed.

It was 10:00 p.m. at night when the break-in was discovered. Nevertheless, the consultant telephoned the system manager at home. When he did, he discovered something else:

- The computer's system manager had known about the break-in for three days, but had not done anything about it. The reason: she feared that the intruder had created numerous holes in their system's security, and was afraid that if she angered the intruder, that person might take revenge by deleting important files or shutting down the system.

In retrospect, this was rather stupid behavior. Allowing the intruder to stay on the system let him collect more passwords from users of the system. The delay also allowed for plenty of time to make yet further modifications to the system. If it was compromised before, it was certainly compromised now!

Leaving the intruder alone also left the company in a precarious legal position. If the intruder used the system to break in anywhere else, the company might be held partially liable in a lawsuit because they left the intruder with free run of the compromised system.

So, what should the system manager have done when she first discovered the break-in? Basically, the same thing as what the outside consultant did: take a snapshot of the system to tape or another disk, isolate the system, and then investigate. If the staff was worried about some significant files being damaged, they should have done a complete backup right away to preserve whatever they could. If the system had been booby-trapped and a power failure occurred, they would have lost everything as surely as if they had shut down the system themselves.

The case above is typical of many break-ins that have occurred in 1994 and 1995. The attackers have access to one of many "toolkits" used to break into systems, install password sniffers, and alter system programs to hide their presence. Many of the users of these toolkits are quite ignorant of how they work. Some are even unfamiliar with UNIX: we have heard many stories of monitored systems compromised with these sophisticated toolkits, only to result in the intruders attempting to use DOS commands to look at files!

24.1 Prelude |  | 24.3 The Log Files: Discovering an Intruder's Tracks